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ABSTRACT 



The Primary Mental Health Project (PMHP) is a 



research-based, selective program implemented by the Jefferson County Public 
Schools. The goal of the program is to enhance learning and other 
school-related competencies. Key structural components of the program include 
a focus on young children, early screening and selection, use of 
paraprofessionals to provide direct services to children, and ongoing program 
evaluation. A management -oriented evaluation approach was used in the 
evaluation of PMHP being used in 12 elementary schools. Both treatment and 
comparison groups were evenly distributed in terms of race and gender. The 
Teacher-Child Rating Scale was used as a pre- and post-test for the 
participating students in the treatment schools only. The central measures 
were related to task orientation, behavior control, assertiveness, and peer 
social. Statistically significant differences were found in the pre- and 
post-test analysis at the district and at the specific domain of behavior 
control. Gains were also noted at most of the individual schools. The 
analyses recommend use of a treatment versus comparison group pre -post -test 
design at the student level. (JDM) 
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PRIMARY MENTAL HEALTH PROJECT (PMHP) 
Research-Based Intervention 



Introduction 

Jefferson County Public Schools (JCPS) is the 26th largest school district in the United 
States. The school district serves more than 96,000 students from preschool to grade 12. JCPS 
has a vision for long-term student achievement. The vision entitled "Beyond 2000" was designed 
to assure that every student will acquire the fundamental academic and life skills necessary for 
success in the classroom and workplace. JCPS vision commits the school system to educate each 
student to the highest academic standards. 

In October 1999, Project SHIELD (Supporting Healthy Individuals and Environments for 
Life Development) received nearly $3,000,000 from a consortium of federal agencies 
(Department of Education, Office of Juvenile Justice and Delinquency Prevention, and Center 
for Mental Health Services) as part of a Safe Schools/Healthy Students Federal Initiative. The 
award will provide three years of funding (nearly $9,000,000) to Jefferson County Public 
Schools (JCPS). 

Project SHIELD aims to provide students and schools with enhanced infrastructure and 
comprehensive prevention and early intervention, through education, mental health, and social 
services that promote healthy childhood development and prevent violence, alcohol and other 
drug abuse. These services target the development of social skills and emotional resilience 
nsc6SS2.ry for youth to avoid violent behavior and drug use, along with establishing safe, 
disciplined, and drug free areas within school environments. 
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Program Description 



The Primary Mental Health Project (PMHP) is a research-based, selective program. This 
early detection and prevention program for preschool and primary grades is being implemented 
by JCPS. It is a nationally recognized model out of Rochester, New York that has been 
replicated in over 200 cities since 1957. 

The key structural components of the program are: (a) focus on young children, (b) early 
screening and selection, (c) use of paraprofessionals to provide direct services to children, and 
(d) ongoing program evaluation. The population targeted for the program are K-3 students who 
are experiencing school adjustment difficulties. The goal of the program is to enhance learning 
and other school-related competencies such as attendance and behavior. 

All K-3 students are screened by having the teachers complete a 12-item survey on each 
of the students. Students who score between the 15‘^ and 30* percentile are considered for the 
program. Once permission is obtained from the parent, the students are enrolled in the program. 

Each student is seen individually by the child associate (paraprofessional) using non- 
directive play strategies for 30 to 45 minutes each week. The program lasts for 14 sessions. 

The child associate works with the student to deal with school adjustment issues and build the 
student’s competencies. A school psychologist provides weekly supervision to the child 
associate. 

Student progress and the effectiveness of the program are measured using a 
pretest/posttest model where the teachers complete a 32-item survey before and after the 
program. 
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Evaluation Model 



The Management-Oriented Evaluation Approach 

The management-oriented evaluation approach (Worthen, Sanders, & Fitzpatrick, 1997) 
was used in the evaluation of the PMHP. According to Stufflebeam (1983; Stufflebeam & 
Shinkfield, 1985), the evaluation is a process of delineating, obtaining, and providing useful 
information forjudging decision alternatives. The Context, Input, Process, and Product (CIPP) 
Evaluation has different objectives, methods, and relation to decision making in the change 
process depending on the type of evaluation emphasis. 

The management-oriented rationale is that the evaluative information is an essential part 
of good decision-making and that the evaluator can be most effective by serving 
administrators, policy makers, boards, practitioners, and others who need good evaluative 
information (Worthen et al., 1997, p. 97). 

Campbell (1969) seminal article on reform as experiments is germane to this evaluation. 
Today, 30 years later, many ameliorative programs terminate with no interpretable evaluation. 
The good intentions of educational administrators are not enough. Establishing social indicators, 
databanks, and management information systems (MIS) is not enough. As Campbell (1969) 
argues, administrators are sometimes so committed in advance to the efficacy of the reform, that 
cannot afford a honest evaluation. Capitalizing on regression, grateful testimonials, and 
confounding selection and treatment are the major strategies to bias the analysis. 
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Method 



Participants 

Twelve elementary schools in JCPS are currently participating in the PMHP. Table 1 
shows the name of the schools participating in the program. 

Table 1 

Elementary Schools Participating in the PMHP (N = 12) 

Name 

Atkinson 
Cochran 
Crums Lane 
Engelhard 
Frayser 

Breckinridge-Franklin 

Hazelwood 

Jacob 

Roosevelt-Perry 

Rutherford 

Semple 

Shelby 
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In the District, about 1,522 students were tested using the AML instrument. At the 
student level, a total of 271 students participated in the program. From this total, approximately 
230 students took the pre- and the posttest. Table 2 presents the socio-demographic 
characteristics of the students that took the AML Behavior Rating Scale. Both treatment and 
comparison groups are more or less equally distributed in terms of race and gender. Table 3 
displays the results of the AML Behavior Rating Scale instrument. As it can be observed, the 
treatment group had a higher need than the control group in the three dimension as well as in the 
total. 

Table 2 

Profile of Participating Students 



Group 


N 


Race 


Gender 


District 


1522 


47.4 Black 
45.6% White 
7% Other 


47.4% Female 
52.7% Male 


Comparison 


1386 


47.8% Black 
45.5% White 
6.7% Other 


47.6% Female 
52.4% Male 


Treatment 


230 


44.1% Black 
47.1% White 
8.8% Other 


44.9% Female 
55.1% Male 
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Table 3 



Ratine Scale 


Group n 


Pre-Test Mean 


SD 


t-value 


Comparison group 1,381 








A raw score 


9.41 


4.66 


3.46* 


M raw score 


8.50 


3.72 


3.87* 


L raw score 


10.78 


4.94 


4.08* 


Total raw score 


28.65 


11.34 


4.29* 


Treatment group 136 








A raw score 


10.87 


4.77 




M raw score 


9.79 


3.46 




L raw score 


12.60 


5.01 




Total raw score 


32.99 


10.37 





Note : 

All p values < .001 
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Instrumentation 



In general, quantitative measures will be based on already established data collection 
mechanism of the county under examination. Data will come from the program director and from 
the Management Information System (MIS) of the county. Then, the evaluator will place the 
information into the Statistical Package for the Social Sciences (SPSS) through the creation of a 
data file. 

The AML Behavior Rating Scale was used to measure most of the students in the primary 
program of the school district under study. The instrument has a long tradition and established 
validity and reliability. Raw and percentile scores are recorded in the instrument. For each raw or 
percentile score, there is an individual score for Acting Out (A), Moody (M), and Learning 
Difficulties (L). 

The Teacher-Child Rating Scale was used as a pre- and posttest measure for the 
participating students in the treatment schools only. The central measures were related to (a) 
task orientation, (b) behavior control, (c) assertiveness, and (d) peer social. These measures will 
become outcome criteria for establishing success of the program at the school level. 

Data Analysis & Procedures 

As mentioned previously, for the quantitative dimension of this evaluation study, a 
descriptive and comparison design will be used (Gall, Borg, & Gall, 1996; Winer,Brown, & 
Michels, 1991). All data was entered and analyzed using the Statistical Package for the Social 
Sciences (SPSS), version 10.0. 
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Findings 

Statistically significant differences were found in the pre- and posttest analysis at the 
district and at the school level. A graphical representation captures the impact of the program at 
the district level in the four critical domains assessed in the Teacher-Child Rating Scale. 



Primary Mental Health Project 2000-2001 
District Data (N = 230) 



CO 

o 

CO 




Task Orientation Behavior Control Assertiveness Peer Social 

Dimensions 



□ Pretest ■ Posttest 
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Table 4 displays the results of this analysis in the specific domain of task orientation. It 
shows the pre-test and posttest measures and their statistically significant t-value at each of the 
participating schools. 

Table 4 

Elementary Schools Participating in the PMHP IN = 12) 



Name 


Pretest Score 


Posttest Score 


t-Ratio 


Atkinson 


18.80 


21.22 


1.96 


Cochran 


19.26 


23.26 


1.99 


Crums Lane 


18.88 


24.08 


4.15* 


Engelhard 


17.70 


18.51 


.62 


Frayser 


18.86 


23.15 


3.40* 


Breckinridge-Franklin 18.91 


21.32 


1.96 


Hazelwood 


20.44 


22.59 


1.5 


Jacob 


19.50 


22.21 


1.24 


Roosevelt-Perry 


21.23 


22.68 


1.66 


Rutherford 


19.90 


21.11 


.65 


Semple 


21.04 


24.09 


2.40* 


Shelby 


18.00 


19.88 


1.52 


District 


1^0 / 


01 O'? 
^ 1 


6.43* 



p < .05 
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Statistically significant differences were found in the pre- and posttest analysis at the 
district and at the school level in the specific domain of behavior control. Table 5 displays the 
results of this analysis. 



Table 5 

Elementary Schools Participating in the PMHP (N = 12) 



Name 


Pretest Score 


Posttest Score 


t-Ratio 


Atkinson 


19.92 


23 


2.98* 


Cochran 


21.32 


24.20 


2.47* 


Crums Lane 


21.71 


25.06 


2.74* 


Engelhard 


23.17 


23.69 


.44 


Frayser 


23.55 


26.12 


2.56* 


Breckinridge-Franklin 22.7 


23.39 


.57 


Hazelwood 


21.52 


23.22 


1.14 


Jacob 


20.72 


25.72 


1.9 


Roosevelt-Perry 


23.27 


23.02 


.2 


Rutherford 


26.19 


25.40 


.66 


Semple 


23.91 


26.26 


2.23* 


Shelby 


22.72 


23.97 


.9 


District 


ZZ.DH 


0/1 O^ 


4.78* 



p < .05 
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Statistically significant differences were found in the pre- and posttest analysis at the 
district and at the school level. Table 6 displays the results of this analysis in the specific domain 
of assertiveness. 

Table 6 

Elementary Schools Participatine in the PMHP (N = 12') 



Name 


Pretest Score 


Posttest Score 


t-Ratio 


Atkinson 


24.88 


27.59 


2.99* 


Cochran 


24.16 


26.04 


1.56 


Crums Lane 


24.46 


27.11 


3.06* 


Engelhard 


21.78 


22.40 


.84 


Frayser 


22.59 


27.59 


6.04* 


Breckinridge-Franklin 24. 18 


25.89 


1.55 


Hazelwood 


25.44 


28.19 


2.19* 


Jacob 


25.11 


23.97 


1.11 


Roosevelt-Perry 


26 


28.8 


2.76* 


Rutherford 


22.9 


25.22 


2.43* 


Semple 


26.13 


27.73 


1.27 


Shelby 


25.44 


28.73 


2.68* 


District 


A AC 


<a< 


7.75* 



p < . 05 
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Statistically significant differences were found in the pre- and posttest analysis at the 
district and at the school level. Table 7 displays the results of this analysis in the specific domain 
of peer social. 



Table 7 

Elementary Schools Participating in the PMHP (N = 12) 



Name 


Pretest Score 


Posttest Score 


t-Ratio 


Atkinson 


24.64 


26.31 


1.65 


Cochran 


23.26 


25.61 


1.51 


Crums Lane 


22.92 


25.82 


2.30* 


Engelhard 


23.61 


23.80 


.2 


Frayser 


26.86 


30.05 


2.82* 


Breckinridge-Franklin 27.14 


27.55 


.40 


Hazelwood 


26.41 


26.01 


.58 


Jacob 


23.67 


26.96 


1.3 


Roosevelt-Perry 


27 


28.80 


1.52 


Rutherford 


27.24 


28.87 


1.25 


Semple 


26.48 


29.38 


2.53* 


Shelby 


29 


28.75 


.19 


District 


25.75 




4.36* 



p < .05 
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Discussion 



The Primary Mental Health Project (PMHP) is a research-based, selective program. This 
early detection and prevention program for preschool and primary grades is being implemented 
by JCPS. The Teacher-Child Rating Scale was used as a pre- and posttest measure for the 
participating students in the treatment schools only. The central measures were related to (a) 
task orientation, (b) behavior control, (c) assertiveness, and (d) peer social. These measures 
became outcome criteria for establishing success of the program at the district and at the school 
level. As a District, the gains on the four factors on the pretest/posttest measure were statistically 
significant at the .001 alpha level. Gains were also noted at most of the individual schools. 

Recommendations for Future Research 



To measure the effects of the program on non-cognitive and cognitive measures, it is 
recommended to use a treatment versus comparison group pre-posttest design at the student level 
in factors such as: (a) absences/attendance rate, (b) tardies, (c) scores on the Stanford Diagnostic 
Reading Test, (d) scores on the Stanford Diagnostic Mathematics Test, (e) referrals for ECE 
assessments and (f) subsequent ECE placements. 
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